%  Copyright (C) 2002 Regents of the University of Michigan, portions used with permission 
%  For more information, see http://csem.engin.umich.edu/tools/swmf
\section{Quick Start \label{section:quick_start}}

The installation instructions are described in the README.md file.
To keep this user manual more up-to-date and consistent,
the README.md file is quoted verbatim below.

\verbinput{../../README.md}

\subsection{Installing the code}

In SWMF the components are installed and uninstalled automatically 
at the same time as the framework is installed.
The Makefile.def and Makefile.conf make files in GM/BATSRUS will 
simply include the files with the same name in the SWMF directory.
The rest of this subsection describes the installation procedure
for the stand alone code.
 
The installation requires the selection of a specific Fortran compiler. 
For many platforms used by CSEM and CRASH, the script can figure out 
the default settings all by itself. 
On these systems, the code can be installed simply by running the command
\begin{verbatim}
  Config.pl -install
\end{verbatim}
in the main directory. This
\begin{itemize}
\item creates {\tt Makefile.def} with the correct 
      absolute path to the base directory, 
\item creates {\tt Makefile.conf} that contains the operating 
      system and compiler specific part of the Makefile
\item creates the compiler specific {\tt Makefile.RULES} from
      {\tt Makefile.RULES.all} in the various source directories,
\item attempts to create a symbolic link {\tt data} to {\tt SWMF\_data/GM/BATSRUS/data}
\item attempts to create a symbolic link {\tt dataCRASH} to {\tt CRASH\_data}
\item executes {\tt make install} which does further
      installation steps specific to \BATSRUS. 
\end{itemize}
If the platform/machine is not listed at the beginning of the
\begin{verbatim}
  share/Scripts/Config.pl
\end{verbatim}
script, or a non-default compiler is desired, then the compiler 
must be specified explicitly with the {\tt -compiler} flag. Type 
\begin{verbatim}
  Config.pl -compiler
\end{verbatim}
to see available choices. Then install the code with the selected compiler,
for example 
\begin{verbatim}
  Config.pl -install -compiler=gfortran
\end{verbatim}
On a Linux system this will copy the
\begin{verbatim}
  share/build/Makefile.Linux.gfortran
\end{verbatim}
file into Makefile.conf. 

In case your platform and/or compiler is not supported yet, 
you will have to create a
\begin{verbatim}
  share/build/Makefile.OS.COMPILER
\end{verbatim}
file where 'OS' is the name of the operating system returned
by the Unix command {\tt uname}, while 'COMPILER' is the name
of the F90 compiler used or something closely related to that.

To uninstall \BATSRUS\ type
\begin{verbatim}
  Config.pl -uninstall
\end{verbatim}
If the uninstallation fails (this can happen if some makefiles are missing)
do reinstallation with
\begin{verbatim}
  Config.pl -install
\end{verbatim}
and then try uninstalling the code again.
When \BATSRUS\ is installed, its configuration can be checked with
\begin{verbatim}
  Config.pl
\end{verbatim}
with no arguments. 
To get a complete description of the usage of the 
{\tt Config.pl} script type
\begin{verbatim}
  Config.pl -h
\end{verbatim}

%^CFG IF TESTING BEGIN
\subsection{Testing the code}

The fastest way of testing the installed code is to run test problems.
The full \BATSRUS\ test suite can be run with
\begin{verbatim}
  make -j test NP=4
\end{verbatim}
on a machine that allows both compilation and parallel execution. 
These tests are typically run on 1 to 8 processors, the default is 2.
If only serial execution is available, the tests can be run serially with
\begin{verbatim}
  make test MPIRUN=
\end{verbatim}
Running the complete test suite can take anywhere from one to several
hours depending on the speed and number of processors, as well as on
the speed of the Fortran compiler. The tests are run in the 
{\tt run\_test\_*} directories, one after the other.

A test is successful if the results agree with the reference solution
to the required accuracy. This is checked by a script, typically
share/Scripts/DiffNum.pl, that compares the output with the reference
solution.  The test passes if the resulting difference file is empty.
See Makefile.test for details.

The full set of available tests is shown by 
\begin{verbatim}
  make test_help
\end{verbatim}
A specific test can be run by typing, for example,
\begin{verbatim}
  make -j test_shocktube
\end{verbatim}
This will do a simple shock tube test that takes a few minutes to complete. 
Like most other tests, this one consists of 4 stages:
\begin{enumerate}
\item make -j test\_shocktube\_compile: configuration and compilation
\item make test\_shocktube\_rundir:  create run directory with input files
\item make test\_shocktube\_run:     run the test
\item make test\_shocktube\_check:   check the results
\end{enumerate}
These stages can be done separately. For example the code can be configured 
and compiled and the run directory can be created on the head node of 
a super computer, and then the run phase can be done on a compute node, 
and the results can be checked on the head node again.
The reference solution of the shock tube test is stored in 
\begin{verbatim}
  Param/SHOCKTUBE/TestOutput
\end{verbatim}
and the difference file will be
\begin{verbatim}
  test_shocktube.diff
\end{verbatim}

%^CFG END TESTING

\subsection{Configuring \BATSRUS}

The first step is selecting the equation and user modules. For example 
\begin{verbatim}
  Config.pl -e=Hd -u=Waves
\end{verbatim}
will select the hydrodynamics equation module and the 'waves' user module
that can perturb variables with various waves, which is useful for tests.
This is essentially the same as executing
\begin{verbatim}
  cp srcEquation/ModEquationHd.f90 src/ModEquation.f90
  cp srcUser/ModUserWaves.f90 src/ModUser.f90
\end{verbatim}
but using Config.pl is simpler and safer, because the previous user module
is saved into {\tt src/ModUser.f90.safe} in case it was replaced accidentally.
The available equation and user modules can be listed with
\begin{verbatim}
  Config.pl -e -u
\end{verbatim}
New equation and user modules can be easily added into the srcEquation/
and srcUser/ directories, respectively.

For some equation modules the number of materials and/or wave groups can be set, 
for example, 
\begin{verbatim}
  Config.pl -e=Crash -nMaterial=3 -nWave=30
\end{verbatim}
will select the CRASH user module and set the number of materials to 3 and the 
number of radiation energy groups to 30.

The adaptive grid parameters can be set, for example, as
\begin{verbatim}
  Config.pl -g=4,4,4 -ng=2
\end{verbatim}
This will create 3D grid blocks consisting of $4\times 4\times 4$ cells
with 2 ghost cell layers (set by -ng).
Note that the number of grid blocks should
be set in the PARAM.in file with the \#GRIDBLOCK and \#GRIDBLOCKALL commands.
The number of cells per
block, however, cannot be changed during run time, which helps the compiler
to properly optimize the inner loops on the grid cell indexes.

If the fifth order scheme is to be used, it requires three ghost cells
and at least 6 cells in all used dimension:
\begin{verbatim}
  Config.pl -g=6,6,6 -ng=3
\end{verbatim}
The grid can also be 2 or even 1 dimensional. For 2D grids the number
of cells in the third dimension is 1, for example,
\begin{verbatim}
  Config.pl -g=10,10,1 -ng=3
\end{verbatim}
will $10\times 10$ grid cells with 3 ghost cell layers in 2 spatial dimensions.
For 1D grids both the second and third numbers are 1, e.g., 
\begin{verbatim}
  Config.pl -g=100,1,1 -ng=2
\end{verbatim}
The current grid settings can be inquired with
\begin{verbatim}
  Config.pl -g
\end{verbatim}
which shows it in the same compact format. A more verbose and complete
configuration information can be obtained with
\begin{verbatim}
  Config.pl -s
\end{verbatim}

\subsection{Compilation options}

Check the SWMF test page for the currently used/recommended 
compiler versions on the various machines. On many systems the
{\tt module} command can be used to see the currently loaded
compilers ({\tt module list}), the available compiler versions
({\tt module avail}) and to unload old and load new versions
({\tt module unload SOMETHING} and {\tt module load SOMETHINGELSE}).

The optimization level can be set, for example, with
\begin{verbatim}
  Config.pl -O2
\end{verbatim}
For debugging it is best to use
\begin{verbatim}
  Config.pl -O0 -debug
\end{verbatim}
The NAG Fortran compiler is especially powerful for debugging, 
as it can check for uninitialized real variables and it provides 
line numbers for any failure of the code.
In some cases the MPI library does not allow the error messages to show up
on the terminal. If the problem can be reproduced with a serial run, 
it is best to set
\begin{verbatim}
  Config.pl -O0 -debug -nompi -noopenmp
\end{verbatim}
The {\tt -nompi} flag will compile the util/NOMPI library and 
modifies {\tt Makefile.conf} so that the code is linked with
the NOMPI library instead of the real MPI library. 
The code compiled with the NOMPI library can only be
run in serial mode, however, debugging can become much easier.
The {\tt -noopenmp} flag disables compiling the OpenMP directives
and the use of the OpenMP library for multi-threaded runs.

To undo these settings, for production runs, use 
\begin{verbatim}
  make clean
  Config.pl -O4 -nodebug -mpi -openmp
\end{verbatim}
Note that {\tt make clean} is necessary when the optimization level 
or debugging flags are changed. If only the NOMPI is replaced with 
the usual MPI library (or the other way around), it is sufficient to
delete the executable, e.g. src/BATSRUS.exe, and simply relink the code
using {\tt make}.

The code can be compiled with single precision using {\tt Config.pl -single},
but this is neither recommended nor supported. On 64-bit platform the default
double precision executes almost as fast as the single precision, and the 
enhanced arithmetic accuracy is required by many algorithms.  
The single precision option wiil likely be removed in the future.

\subsection{Compilation}

Once the compilation options are set, compile \BATSRUS\ with
\begin{verbatim}
  make -j
\end{verbatim}
in the main directory. This will produce the executable 
{\tt src/BATSRUS.exe}. The CRASH code, which is \BATSRUS\
compiled together with the util/CRASH library, can be created with
\begin{verbatim}
  make -j CRASH
\end{verbatim}
and the executable will be {\tt src/CRASH.exe}. 

For post-processing IDL output, the executable {\tt src/PostIDL.exe}
have to be compiled with 
\begin{verbatim}
  make PIDL
\end{verbatim}
The multiple snapshots included in a single IDL output file can be 
manipulated with the {\tt src/SNAPSHOT.exe} code, 
which can be compiled with
\begin{verbatim}
  make SNAPSHOT
\end{verbatim}

\subsection{Creating a run directory}

The next step is to create a run directory with
\begin{verbatim}
  make rundir
\end{verbatim}
This will create a directory named {\tt run/}. This can be renamed
or even moved to another disk (e.g. the scratch disk of a super
computer), if necessary. The name of the run directory can also
be specified like this 
\begin{verbatim}
  make rundir RUNDIR=/scratch1/MYSELF/run15
\end{verbatim}
Note that an absolute path is used. 
In some cases it is useful to create a run directory that mimics
the use of \BATSRUS\ as some specific component of the SWMF. 
For example, for the solar corona component use
\begin{verbatim}
  make rundir COMPONENT=SC
\end{verbatim}
This will create a {\tt run/SC} subdirectory instead of the usual
{\tt run/GM} subdirectory.

The freshly created run directory contains several files, symbolic links, 
and directories. The most important one is a link to the executable BATSRUS.exe. 
If the run directory is created with
\begin{verbatim}
  make rundir DEFAULT_EXE=CRASH.exe
\end{verbatim}
then it will contain a link to the CRASH.exe code. There are also links to
the PostIDL.exe and PostSPH.exe codes used for post-processing the output.
It is often a good idea to remove the links and copy the executables into
the run directory. This allows recompiling the code without having a possibly
unwanted effect on the run directory, or a submitted run that is still in 
the job queue.

The IO2 subdirectory will contain the output files, while the restartIN and restartOUT
subdirectories are used for input and output restart files, respectively.
The PostProc.pl, pIDL, pTEC, and Restart.pl scripts are copied into the 
run directory to allow processing the output files and restarting the code.
The link to the PostIDL.exe code is for postprocessing.

If the machine name (disregarding numbers at the end of the name) matches 
one of the job files in 
\begin{verbatim}
  share/JobScripts
\end{verbatim}
then the job file will be copied
into the run directory. For example, on {\tt pfe8} (one of the head nodes 
of Pleiades), the {\tt job.pfe} job script file and the {\tt qsub.pfe.pl}
and {\tt watch.pfe.pl} scripts will be copied. The job script file serves as a
template only. It has to be edited and modified before submitting a job.
On Pleiades the {\tt qsub.pfe.pl} script can be used to submit jobs for
multiple node types, see the help message (-h) for details.

\subsection{Creating input files}

The run directory also contains a "default" PARAM.in file, but this should always
be replaced with another file, in practice.
The link to the BATSRUS/Param directory is provided, so that the files in
this directory can be copied into PARAM.in as a starting point. 
Some of the files contain small segments of the input file, e.g. the grid
description for some application. These segments can also be included
into PARAM.in with the {\tt \#INCLUDE} command.

It is important to check the correctness of the input file, especially before submitting
large jobs on a super computer. In the main \BATSRUS\ directory type
\begin{verbatim}
  CheckParam.pl
\end{verbatim}
which will check {\tt run/PARAM.in}. The number of processors to be used and
the name of the input parameter file can also be specified, for example
\begin{verbatim}
  CheckParam.pl -n=256 run_large/PARAM.in
\end{verbatim}

Another way to construct and check the PARAM.in file is to run the parameter editor
(this can be done best on your local desktop or laptop machine):
\begin{verbatim}
  share/Scripts/ParamEditor.pl
\end{verbatim}
This will open {\tt run/PARAM.in} in a web browser with a user interface.
The name of the parameter file can also be specified, e.g., 
\begin{verbatim}
  share/Scripts/ParamEditor.pl run_large/PARAM.in
\end{verbatim}
The parameter editor provides an integrated user interface with manual, 
editing, and checking options.

The PARAM.in file is always required for a run. Several applications read additional files,
for example for initialization, boundary condition, satellite trajectories to extract data, 
and in case of a restarted run, restart files. These all have to be in place before
the code is run. The parameter checking also checks for the existence of many of these
input files.

\subsection{Running the code}

Now you are ready to run the code. For an interactive run 
on 8 processors with 4 threads (if the code was compiled with an MPI
library and OpenMP enabled) type
\begin{verbatim}
  cd run
  export OMP_NUM_THREADS=4; mpiexec -np 8 BATSRUS.exe # bash, ksh, zsh
  setenv OMP_NUM_THREADS 4; mpiexec -np 8 BATSRUS.exe # csh, tcsh
\end{verbatim}
It is often a good idea to save the output and error messages into a file,
both for runs done interactively and in the background:
\begin{verbatim}
  mpiexec -np 8 BATSRUS.exe |& tee runlog
  mpiexec -np 8 BATSRUS.exe >& runlog &
\end{verbatim}
Using {\tt runlog} as the filename allows the {\tt PostProc.pl} script
to find it and collect it together with other output files.

On many machines you have to submit the run to a batch queue.
First edit the appropriate job script file and then submit the job,
for example on Pleiades
\begin{verbatim}
  cd run
  emacs job.pfe
  ./qsub.pfe.pl job.pfe Run8
\end{verbatim}
The content of the job scripts and submission commands depend on
the queuing system. 

\subsection{Restarting the run}

There are several reasons for restarting a run
\begin{itemize}
\item a run may be too long for a single job submission,
\item a run may fail for some reason, and it can be restarted with different parameters,
\item a parameter study is done with varied parameters starting from the same state,
\item debugging some problem that happens a few steps after the restart.
\end{itemize}
The {\tt Restart.pl} script makes it quite easy to restart a run. Assuming that there are
restart files saved, simply type
\begin{verbatim}
  Restart.pl
\end{verbatim}
in the run directory. This will move the files from the {\tt restartOUT}
directory into a restart directory tree, and the files in the restart
directory tree will be linked to the {\tt restartIN} directory.
The default name of the restart directory tree is based on the simulation time
for time accurate runs, or the number of iterations for steady state runs. 
For example, if the restart files 
were saved at 6 hours of simulation time, the restart directory will be named
\begin{verbatim}
  RESTART_t006.00h
\end{verbatim}
The name of the restart tree can also be specified, e.g., 
\begin{verbatim}
  Restart.pl RESTART_debug
\end{verbatim}
It is also possible to select an already existing tree, e.g. 
something that was saved by PostProc.pl (see next subsection):
\begin{verbatim}
  Restart.pl -i NewResults/RESTART
\end{verbatim}
By default the restart files overwrite each other, however, 
the {\tt Restart.pl} script can create multiple 
trees while the code is running, by simply checking if there 
are new restart files in restartOUT and moving those into 
a tree. For example
\begin{verbatim}
  Restart.pl -r=600 -t=h &
\end{verbatim}
will check for new restart files every 600 seconds (wall clock time), 
and the restart trees will be named using hours as time units.
To read about more options, type
\begin{verbatim}
  Restart.pl -h
\end{verbatim}

\subsection{Postprocessing}

After the code has successfully run (possibly with multiple restarts) 
the output files found in {\tt run/IO2} need some post processing before 
they can be visualized. The most convenient way to do this is to run
\begin{verbatim}
  PostProc.pl
\end{verbatim}
in the run directory. This will process both IDL and Tecplot output files.
For long runs it is a good idea to periodically post process the output, e.g.,
\begin{verbatim}
  PostProc.pl -r=600 >& PostProc.log &
\end{verbatim}
will post process the output every 600 seconds, and the various
output (and possibly error) messages will be collected into 
the {\tt PostProc.log} file. 

When the run is finished, the IDL output files can be merged into 
a single ``movie'' file with the {\tt -M} switch, 
while the logfiles and satellite output files from multiple restarts 
can be concatenated using the {\tt -cat} swithc:
\begin{verbatim}
  PostProc.pl -M -cat
\end{verbatim}
When the run is complete, it is a very good idea to create an output directory tree
that contains the {\tt PARAM.in} file, the {\tt runlog} file, the plot files as well
as restart files together. This can be done by providing the name of the directory
tree as an argument:
\begin{verbatim}
  PostProc.pl -M -cat RESULTS/latest
\end{verbatim}
When there are a huge number of output files to process, 
the post processing can become slow. The post processing
of the IDL files can be done in parallel on multicore
machines using the {\tt -n} switch
\begin{verbatim}
  PostProc.pl -n=4
\end{verbatim}
This can also be combined with the {\tt -r} switch for continuous 
post-processing when the serial post processing cannot keep up with
the amount of data produced by the running job. The complete set
of options can be obtained with
\begin{verbatim}
  PostProc.pl -h
\end{verbatim}
There are several lower level scripts that can be used directly if desired. 
For example the IDL output files can be processed with
\begin{verbatim}
  pIDL
\end{verbatim}
This command will produce the new {\tt run/IO2/*.out}
files, and delete the {\tt run/IO2/*.idl} files
and the corresponding header files {\tt run/IO2/*.h}.
In case you want to keep the raw data, e.g. to be safe, type
\begin{verbatim}
  pIDL -k
\end{verbatim}
For complete usage information type {\tt pIDL -h}.

The Tecplot files are processed with the {\tt pTEC} script.
See {\tt pTEC -h} for options.

Visualization of the resulting output files with IDL and TecPlot
will be described in detail in sections \ref{section:idl_visualization}
and \ref{section:tecplot_visualization}, respectively.

\subsection{Recompilation and clean up}

If you change some source files, e.g. the user module {\tt src/ModUser.f90},
the code can be simply recompiled with the {\tt make} command. 
However, if the compiler flags or the precision are changed, you need to clean the
object files with
\begin{verbatim}
  make clean
\end{verbatim}
before recompilation. This will remove all the object files and forces recompilation.

To delete all files created during installation and compilation, use
\begin{verbatim}
  Config.pl -uninstall
\end{verbatim}
In addition to the object files, 
this command will remove the executables {\tt src/*.exe}, 
the libraries {\tt src*/*.a} and all other temporary files.

For a complete list of possible make targets type
\begin{verbatim}
  make help
\end{verbatim}
